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those tools provide users with the unique functional requirements associated 
with using data in the context of decision making. MPS has a sophisticated 
and rich information environment with both an operational database and a 
relational data warehouse. In this study, the Quality School Portfolio (QSP) 
was introduced as a local database tool to increase schools' capacity for 
using data to inform decisions. The first section of the report presents an 
overview of the MPS information technology infrastructure. QSP must interface 
with and compete with the technology already in place in the MPS. The second 
section summarizes a four-dimensional framework that is used to describe how 
QSP compares with and contrasts with other information tools supported by 
MPS. The third section evaluates QSP and other MPS tools on the basis of 
functions considered important to the process of data-driven decision making 
in the MPS. The comparison of QSP to Brio Insight and Excel shows that QSP 
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QSP AND THE MPS INFORMATION SYSTEM 



Jeffery Watson 



Overview 

This paper summarizes an analysis in which disparate information tools and 
systems within Milwaukee Public Schools were compared and contrasted from the 
viewpoint of whether or not those tools provide users with an the unique functional 
requirements associated with using data within the context of decision making. The goal 
of this analysis is to better understand the way in which any single piece of technology 
will interact with its counterpart technology. In this sense, the goal is not a straight 
comparison of one tool to another, but to understand how multiple tools can be used 
together as a system to support data-informed decision-making. 

MPS has a sophisticated and rich information environment, in which there are two 
main systems: an operational database and a relational data warehouse (Kimball, 1996). 
An operational database collects student data (attendance, grades, and incident referrals) 
in real time via classroom-based computers. The data warehouse is a relational database 
that contains operational data, as well as other types of data, such as state test results, 
historical data, and demographic data. Whereas the operational system is designed to 
collect real-time data, the data warehouse is designed to support the analysis and 
reporting of data. In conjunction with the district information systems, many schools 
maintain independent databases and tools. Our project introduced QSP as a local database 
tool to increase schools’ capacity for using data to inform decisions. 

This report, organized into three main sections, will provide the reader with a 
sense of the technical and user centered constraints that impact data use within schools. 
The first section, A Review of MPS Technology, presents the reader with an overview of 
MPS’s information technology infrastructure and describes some of the important 
features of that infrastructure. Information flow within the district is also described. This 
section is important to this report’s thesis because QSP must both interface with and 
compete with technology already in place in MPS. The second section. An Information 
Tool Framework, summarizes a four-dimensional framework (Thom, Watson, & Zeyher, 
2001) that is used to describe how QSP compares with and contrasts to other information 
tools supported by MPS. This framework is also useful for describing the overall nature 
of information tools and can be looked at as a high-level review of information systems 
technology. The third section. Analysis of QSP in the context of MPS, evaluates QSP and 
other MPS tools on the basis of functions we consider important to the process of data- 
driven decision-making in MPS. 

A Review of MPS Technology 

It is important to note that MPS has a number of information systems that serve 
the district. This section describes the two most important systems and how information 
flows between them. MPS has adopted a two-prong approach that is typical of business 
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information systems solutions (Figure 1). The School Management System (SMS) is 
responsible for collecting and storing up-to-the-minute information on students. The 
other system, the data warehouse (DW), is responsible for storing long-term information 
and supports the district’s data analysis processes. There are two main entry points for 
data that are of concern. Classroom grades, attendance, and some incident referral data 
enter via SMS, whereas district and state test scores are imported directly into the data 
warehouse. In general, the information that is in SMS is imported to the data warehouse 
by the Technical Services Department. Once data are in the data warehouse, they can be 
exported, analyzed, or viewed via Brio. Insight. 

The School Management System (SMS) is an information system that collects 
attendance data, incident data, and classroom grades in real time. Teachers enter data into 
SMS directly from the classroom and once entered, the classroom data can be queried 
and reported through the SMS software. Many of the reports provided by SMS attempt to 
provide schools with standard and predictable information tables, but ad hoc queries can 
also be created if an administrator needs information that is not already available through 
one of the standard reports. 

In technical terms, the SMS is an operational database, or transactional system, 
which is designed to capture information each time a “transaction” occurs, such as the 
recording of a student’s absence from his fifth hour algebra class. The name itself is a 
throwback to systems that were developed in the 1980s to link a business’s point-of-sales 
(i.e., transactions) data to the company warehouse, shipping, and manufacturing systems 
(Kimball, 1996). There are several characteristics of transactional systems that are 
noteworthy. First, transactional data sets are considered “alive” because the data will 
change over time, usually over a very short period of time. This is true of the SMS 
because data are being added to the system continuously (at least, during school hours). 
The data in the SMS are also subject to being edited by users and database administrators. 
Consider the student who arrives late for his algebra class. The teacher may initially mark 
him as absent, but later change his absence to an unexcused tardy. Thus, anyone trying to 
view fifth-hour attendance data might see slightly different numbers depending on when 
he or she conducted a query. 
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Figure 1. This diagram illustrates how information flows among MPS information systems. Most data 
originate from either the SMS system (grades, attendance, and some discipline data), or from state and 
district test scores datasets. The data warehouse stores data for querying and analysis purposes. 




Another characteristic of transactional systems that holds true for the SMS is that 
performance is optimized by dividing data across a number of data tables so that 
redundancy within and across tables is minimized. This means that transactional data are 
very hard to access, both in terms of knowing where a certain piece of data is stored and 
of being able to pull out large sets of data in a timely manner. In fact, getting data out of a 
transactional system requires the assistance of a database professional, using either a 
custom-written query program, or a prefabricated query program. Therefore, certain data 
queries can be well represented in transactional models, assuming that the data query is 
known ahead of time, does not change over time, and is reasonably small. 

For the above reasons, the SMS is the entry point for much MPS data, especially 
data that are meant to measure students’ daily and even hourly performance, such as 
attendance, classroom grades, and some incident data. Because of the way in which 
transactional systems are designed to collect and maintain real-time data, the SMS is not 
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a good environment for analyzing that data. The reasons for this lie in the fact that data 
are spread out over dozens of data tables. Therefore, extracting data requires intimate 
knowledge of the contents of each data table in the system. Furthermore, SMS is not 
designed to handle the workload associated with district-wide data extraction and 
analysis. Therefore, MPS has implemented a second system, its Data Warehouse, to 
handle data analysis. 

The data warehouse is an approach to information systems that uses a relational 
database management system. This is fundamentally different from a transactional 
database. First, a relational database will contain all of the basic data “facts” within a 
single data table. This table is then associated to other tables that define the important 
dimensions of that data. For instance, the basic fact table contains a list of students 
(identification numbers and names). That basic table would then be connected to other 
tables that contain dimensional information, such as attendance (days enrolled, days 
attended, and percent attended), student demographics (ethnicity, gender, home address, 
etc.), and test data. Some of these dimensions may have sub-dimensions, depending on 
the nature of the data. For instance, test data could be associated with specific types of 
tests and their subscores. 

Another difference between relational and transactional databases is that 
relational databases are stable and do not change over time, as transactional systems do. 
Whereas in a given day, a transactional system may handle thousands or millions of data 
loads (updating the database by adding or changing information), each of which only 
affect a single record, a relational database will receive new or updated data only 
occasionally, but that data load will likely contain millions of records. 

The benefit of a relational database is that it is designed to support querying and 
analysis. For this reason, the Data Warehouse has become the central database for MPS. 
Data are loaded into the Data Warehouse from a variety of sources, including SMS, 
testing vendors, and legacy mainframe systems. In terms of information flow, the Data 
Warehouse is where most MPS data resides until it is needed. 

Since the Data Warehouse is designed for data querying and analysis (i.e., On- 
Line Analytical Processing or OLAP), the process of extracting data is relatively easy and 
very efficient. MPS has chosen to use Brio.Insight (version 6.0) as a front-end tool to 
access the Data Warehouse. Brio.Insight is simply a piece of software that provides users 
with a drag-and-drop interface to the data tables maintained in the data warehouse. Once 
data are extracted, users can also use Brio to sort data, make tables, and create figures 
(e.g., graphs and charts). 

It is also important to note other features of MPS information systems. First, one 
should be aware that the above systems are designed to serve district-wide needs; they are 
not designed to handle data that an individual school might collect on its own (See Figure 
1). Of the six schools we worked with, four committed resources to collecting, managing, 
and maintaining local data. Two schools used Accelerated Learner’s STAR Math and 
Reading software to test students’ individual progress in those subject areas. One other 
school was interested in maintaining more detailed data on student incident referrals. The 



fourth school was interested in maintaining in-house data on incoming students as well as 
on current students; This school devoted personnel to gathering and managing its data. As 
is evident, schools had the need to integrate local data with district data. Indeed, as will 
be discussed later, the need to integrate data that is not in either the Data Warehouse or 
SMS may be a significant feature affecting the overall usefulness of QSP in MPS. 

An Information Tool Framework 

We presented a four-dimensional framework for the comparison and evaluation of 
information tools in an earlier report to the National Institute for Science Education 
(Thom, Watson, & Zehyer, 2001). The framework is intended to be useful in 
understanding how various technologies interact within the context of an educational 
setting, especially when the technologies are designed to accomplish the different goals 
or provide unique functionality. The first dimension, user needs, identifies specific user 
groups and the anticipated user requirements of an information tool or system. The 
second dimension, data characteristics, attempts to illistrate the different ways in which 
data can be treated by an information tool. The third dimension, analytic tools, presents 
the different ways in which data and information can be analyzed. This dimension is 
meant to distinguish among information tools on the basis of how they allow users to 
analyze data. The fourth dimension, technical considerations, identifies a series of 
technical differences that exist in information technology. 

Using The Framework 

These dimensions serve as a framework for understanding similarities as well as 
differences between information tools. Therefore, it may be that one might want to 
compare two or more tools that are designed to be functionally identical. In such a case, 
the goal would be to understand which tool best supports that functionality. Whereas in a 
case where two or more tools are very different from each other, the framework can be 
used to understand how these systems complement each other. The latter is more 
applicable to our experience in MPS. 

It should be noted that QSP has been under development for a relatively short 
period of time and does not have the benefit of a commercial software development 
budget. For these reasons, it may not be fair to compare QSP to other tools that MPS 
personnel may use when working with data. However, it is fair to ask how QSP will 
interact with these tools. It is also fair to ask how likely QSP is to diffuse into MPS as an 
information tool. Understanding this interaction will lead to better understanding of how 
to improve MPS capacity to use data to guide decisions and future research. 
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User needs. This dimension identifies how an information tool meets the needs of 
a particular set of users. In general, there are four main groups, each of which has fairly 
distinct information needs. Table 1 summarizes those needs, as well as identifying the 
time frame within which each group is likely to be operating. Likewise, these groups are 
likely to differ in how they aggregate data across individuals. Members of a school’s staff 
and administration are likely to focus on individual students or small groups of individual 
students when analyzing data, whereas district staff are more likely to report and analyze 
data that have been aggregated across a greater number of students. 

Table 1 



A Summary of Each User Groups ' Needs by Data Characteristics 





Data Characteristics 


Information of Focus 


Average 
Time Frame 


Average Level of 
Aggregation 


User Group 


Students and 
Parents 


Classroom measures. 
Standardized tests 


Daily 


Single individual 


Teachers 


Classroom measures, 
test scores 


Daily 


Multiple individuals, 
small groups 


School 

Administration 


Mark period grades. 
Standardized tests. 
Longitudinal data. 
Local program data. 
Local budget and 
planning data. 
Federal compliance 
measures 


Mark Period 


Individual to school- 
wide 


District 

Administration 


Standardized tests. 
District budget and 
planning. 

Federal compliance 
measures 


Annual 


Schools, district-wide 
groups 



Data characteristics. This dimension refers to the way in which an information 
tool treats data in general. At the basic level, databases will contain a set of records (these 
are like the rows of a spreadsheet), each of which contains fields (columns). However, 
there are also additional characteristics that can improve the way in which data is stored, 
analyzed, or reported. Namely, metadata, which, defined as simply information about 
information, can be very useful in supplying users with definitions of data elements as 
well as information about when the data were created, last updated, and how those data 
possibly relate to other data. Metadata can also be useful in more sophisticated 
information systems for defining ways in which data can be aggregated — for example, 
across time or students. 

Also, data has two characteristics that are dependent on the source, as well as on 
the information tool used to access it. Both characteristics have to do with how one 
aggregates data. Aggregating over time decreases the temporal resolution, such as 
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reporting individual attendance for every semester rather than every week or each day. 
Aggregating across individuals increases the granularity of data — for example, reporting 
the percentage of students that has met a proficiency by school by grade. These 
characteristics are determined mostly by the maimer information systems professionals 
decide to use to collect and store data. 

Analytic structures. This dimension captures the various ways in which users of 
an information tool analyze data. Table 2 summarizes this information in a K- 12 context. 
In general, the components of this dimension vary from simple to complex. The simplest 
form of analyses involves writing queries and reports and, perhaps, averaging over some 
arbitrary group of records, such as school or grade level. Slightly more sophisticated are 



Table 2 

A Summary of Analysis Techniques that Comprise the Analytic Structures Dimension 





Definition 


Example 


Analytic Structure 


Queries and 
Reports 


Generating a set of 
records and sorting those 
records in a meaningful 
way 


Extracting a list of students 
who have yet to meet 8*'’ 
grade proficiencies 


Aggregation 


Summing or averaging 
over a basic unit of 
analysis 


Reporting average WSAS 
test scores by school 


Multi-dimensional 

Graphing 


Supports the graphical 
representation of multi- 
dimensional data 


Bar graphs of a measure 
across grade, gender, 
ethnicity, etc.. . 


Statistical Analysis 


Computational analysis 
that attempts to find 
statistically reliable 
patterns in data 


t-Test, Analysis of Variance, 
Multivariate Analysis of 
Variance, Correlation, 
Regression 


Data Mining, 
Model Forecasting, 
and Prediction 


Automated pattern 
detection and 
mathematical model 
building and testing 


Factor analysis, linear 
systems analysis 



multidimensional graphing techniques that allow the user to create graphical 
representations of data that have many dimensional attributes. For example, in analyzing 
WSAS test scores, one might choose breakdown test scores by gender and ethnicity and 
student reading ability, resulting in a three-dimensional graph. Furthermore, in more 
sophisticated information tools, the graph itself can be interactive, resulting in a dynamic 
representation of /j-dimensions. Statistical analysis is a more complicated analytical 
approach that requires the user to be aware of issues relating to statistics in general (i.e., 
what is variance, reliability, power) and of issues relating to specific statistical measures 
and tests. Finally, some tools are designed to automatically analyze data sets for patterns 
that appear to be statistically reliable. These patterns may lend themselves to building a 
mathematical model that may be used to predict future data. 




8 



9 



Technological considerations. This dimension of the information tool framework 
identifies several issues relating to the technical aspects of information systems. The first 
issue, usability, is a general design aspect that can be applied to all types of tools and 
systems. It is especially important when an end-user tool is being considered because the 
intended users are not likely to be highly skilled in the computer field. Usability is also an 
issue in that it can affect the cost/benefit ratio of a given tool. A system that is hard to 
learn or operate will lead to poorer end-products, while elevating the costs of operation 
(i.e., it requires more time or personnel). 

Another distinction to make when looking at information tools is the 
implementation strategy of the tool. Whether or not a tool is designed as a stand-alone 
product or as a network application will impact the way in which an organization will 
utilize it. Network applications can be further divided into either client/server 
applications or web-based applications. Client/server architecture is the traditional 
application but may have special requirements relating to operating systems, licensing, 
and the site of the license installation. Alternatively, web-based applications are written 
to use web-browser platforms such as Netscape or Internet Explorer as a basis for the 
software. As such, these implementations tend to be platform-independent and more 
flexible in terms of when and where a license can be used. 

Traditional information systems vary in their focus concerning data use. On-line 
transactional processing systems (OLTP) are designed to manipulate single records of 
data, such as updating a student’s attendance for a particular hour. These systems can 
handle an impressive amount of real-time data, but are limited for purposes of data 
analysis. On-line analytical processing (OLAP) systems are designed to compensate for 
this limitation of OLTP systems. Here, analysis is emphasized and accomplished in a 
variety of ways (see discussion above on Analytic Structures). Also related to this issue is 
whether or not an information tool is designed to facilitate the everyday workflow of a 
specific school or district. Workflow management systems attempt to streamline and 
coordinate the process of using data. This aspect of information technology would only 
be present in very large-scale implementation strategies. 

Other issues include performance and system requirements. Both of these issues 
may be specialized, depending on the type of tool being considered. For example, 
Microsoft’s Access is fairly limited in terms of database size. Thus, capacity would be 
considered a relevant consideration if it is expected that the database would exceed the 
limits of the system. Conversely, performance issues only arise during the analysis of 
very large data sets. 



Analysis of QSP in the Context of MPS 

As has been indicated in the previous sections, MPS maintains a sophisticated set 
of information tools and systems. Furthermore, information tools and systems, in general, 
vary along many attributes. The focus of this section is to determine whether QSP will 
complement the technology already in place in MPS and, if so, specify how QSP will 
work within the pre-existing structure of MPS’ s information technology. 
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In order for QSP to successfully diffuse into MPS schools, it must accomplish one 
of two goals. QSP must either provide unique a functionality that is critical to the 
workflow of schools in MPS, or it must provide functionality that is already available but 
at a lesser cost than competing software. Cost can be measured in many ways and, for the 
most part, should not be measured in dollars. While monetary costs may impact the 
decision to buy a piece of software, it tends not to impact users’ daily decisions regarding 
use of the software. Costs are more realistically measured by attributes such as time to 
learn, interoperability, ease of use, and software stability. 

There are three main tools in MPS that are likely to play a significant role in 
schools’ use of data: SMS, Excel, and Brio. Insight. QSP provides a fourth option. To 
summarize briefly, SMS is the interface with the district’s transactional information 
system and provides functionality for data collection, some data extraction, and some 
reporting functionality. Brio. Insight is the district’s query tool for the Data Warehouse, 
which also provides reporting, graphing, and analysis features. Excel is a powerful 
spreadsheet tool that can be used for data management, analysis, and reporting. QSP is 
designed for data management, analysis, and reporting. 

This section is divided into two parts. The first part is concerned with detailing 
how QSP differs from Excel, SMS, and Brio.Insight. The second part is meant to provide 
a comparative evaluation of QSP, Excel, Brio, and SMS in terms of the critical functions 
required in data-driven decision-making. 

The Information Tool Framework detailed in the previous section of this report 
facilitates a comparison and contrast of the four information tools. As expected, these 
tools vary in terms of what fimction(s) they are designed to provide. The framework 
indicates how the attributes of these tools overlap, allowing us to understand how each 
tool interacts with the other three. For example, the two dimensions. User Needs/Groups 
and Analytical Structures Supported, can be used as axes for a graph that represents what 
user group each tool is designed for and what analytical functions they support (Figure 
2). This figure indicates that when it comes to targeting user groups and providing 
analytical support, there is a significant amount of overlap among the four tools. Most of 
this overlap occurs because all four tools target school administration, while providing at 
least some reporting, summary (i.e., aggregation across groups), and graphing 
functionality. However, this graphic also suggests that QSP and Brio.Insight offer more 
extensive functionality than both SMS and Excel. QSP also targets teachers, whereas 
Brio.Insight provides greater analytic capabilities. 
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Figure 2. A visual representation of the user groups targeted by QSP, Excel, SMS, 
and Brio. Insight and the analysis they support. 




Reporting Aggregation Multidimensional Statistical Data 

Graphing Analysis Mining 



Analytical Structures Supported 

Setting the dimensions. User Group and Technical Considerations, orthogonal to 
one another creates a table in which each user group’s technical needs can be specified 
(Table 3). In general, there are five technical aspects that can be considered significant: 
usability, implementation strategy, OLAP vs. OLTP, capacity, and data integration. Table 
3 also details whether or not each tool can meet the minimum requirements for each user 
group. 



Usability is an important design feature because it will impact the everyday use of 
a software tool, as well as the efficacy of its usage. Also, due to work loads and technical 
skill sets, it is likely that teachers and students/parents will have more stringent usability 
requirements than school and district administrators. Administration will in all likelihood 
have support personnel and other resources to help in overcoming usability issues, should 
they arise. Of the four tools, only Brio.Insight provides an interface that is expected to be 
usable on a wide scale by teachers (see Table 3). This is primarily due to the fact that 
Brio allows for three levels of users, administrator, analyst, and consumer, so that once a 
report or analysis has been generated, it can be dynamically viewed (but not necessarily 
edited) by other users. In this way, users with minimal skills can access and explore data 
that is specific to their work domain. 
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Table 3 

Summary of Technical Considerations Across User Groups 





Technical Considerations 


Usabilit; 




Web- 


jased 


Analytical 

Focus 


Capacity 


Data 

Integration 


QSP 


Excel 


SMS 


o 

'C 

CQ 


QSP 


Excel 


SMS 


o 

•C 

PQ 


QSP 


Excel 


SMS 


o 

•C 

PQ 


QSP 


Excel 


SMS 


o 

•C 

PQ 


QSP 


Excel 


SMS 


o 

•C 

PQ 


User Group 


District 

Admin 




V 




V 








V 




V 




V 








V 








V 


School 

Admin 




V 




V 








V 




V 




V 








V 




V 




V 


Teachers 








V 








V 








V 




V 




V 










Students/ 

Parents 


's. 
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Table 3 also indicates that only Brio. Insight uses a web-based implementation 
strategy. This is important for two reasons. First, it supports both Apple and Windows 
users. This is very important at the school level, where many users prefer the Apple 
platform. A web-based implementation also encourages the centralization of data, which 
makes the task of data management more manageable. QSP and Excel are both stand- 
alone applications, requiring the maintenance of data on each computer that is used. SMS 
employs a traditional client-server implementation that is better than a stand-alone 
implementation, but still limiting because users must use computers on which the client 
software has already been installed.* 

The next technical consideration is whether or not the tool supports an analytical 
focus in data use. QSP and Brip provide good support for data analysis because their 
interface designs encourage the user to utilize several graphing functions to analyze data. 
Excel also provides good support for analysis of data through graphing functions, as well 
as through supporting advanced statistical analysis. SMS does little to encourage 
analysis; a query to this system results in a list of students, which would then need to be 
analyzed by some other software package. 

In terms of data capacity, both SMS and Brio can handle the amount of data 
required at the district level, but only Brio is a realistic query tool for exploring data. This 
primarily stems from the fact that SMS is a transactional system, whereas Brio is 
considered an OLAP tool. Both Excel and QSP share similar capacity limits. Excel 
spreadsheets can have no more than 65,536 records (rows) and 256 fields (columns). 



' Web-based software such as Brio also requires locally installed software. This software is downloaded 
when the users access the server through a web-browser, effectively making the server accessible from any 
machine with a Web browser and Internet access. 
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QSP, which is based on Microsoft’s Access database management system, is limited to 
255 or fewer columns. Likewise, QSP is also limited to a total of two gigabytes for each 
database and one gigabyte for each record set. Both Excel and QSP are adequate for 
school and classroom analyses; however, the column limit reduces the effectiveness of 
both software packages for district-wide data sets. For example, a data file containing 
demographic information for MPS will contain over 100,000 records, less than 40 fields, 
and require well over 10 gigabytes of space. Another example is the data set for the 
Wisconsin State Assessment System’s TerraNova test. The data set supplied by the test 
vendor will contain over 300 fields and, as MPS shifts towards annual testing, almost 
100,000 records. 

It should be noted, however, that both Excel and QSP allow the user to select the 
fields to import, which minimizes the negative impact of each tools’ field limit. Both 
import wizards utilize a set of dialogue boxes that guides the user through each step 
required to import data. Excel’s wizard consists of three boxes, each with a “back” button 
that allows the user to move backwards through the process if a mistake is made. QSP 
uses five boxes, none of which have a “back” button (Appendix 1). Excel also supports 
the use of multiple delimiters, as well as an interactive drag-and-drop interface for 
defining field borders. While QSP is not as flexible as Excel in allowing users to 
manipulate multiple delimiters and field borders, it does have an “ignore rest” button. 
This is very helpful in that often the user may only want to import a few of the fields 
contained in a data file. This allows the user to quickly discard unwanted information. 

However, in our experience, successfully importing data from a very large data 
set into QSP is often difficult because the process is prone to errors. The source of these 
errors is unclear, both to the end-user and to us as technical assistance providers. 
Furthermore, the error messages presented by QSP do not provide users with a solution 
path. Instead, the user must try to deduce both the problem and solution. We have found 
that adding a field containing a dummy variable to a data set will sometimes solve import 
errors, but this must be done via another software package (e.g.. Excel) and can be 
problematic if the data set is large. 

The last technical consideration represented in Table 3 is data integration. This 
issue addresses the problem of integrating data from multiple sources, including district 
information systems, school data systems, test vendors, and others. At the current time, it 
is unclear whether Brio can be used to look at local, non-district-based data sets. An 
example of such a database would be a collection of behavioral data, or test data that is 
generated by an in-school assessment system. As Table 3 indicates, QSP and Excel are 
more likely to be useful at the school and classroom level than Brio or SMS. QSP has an 
additional edge over Excel in that it will automatically join data sets through students’ 
identification numbers. 

As stated at the beginning of this section, we predict that QSP must meet one of 
two conditions if it is to successfully diffuse into MPS schools. QSP must either provide 
unique and necessary functionality, or it must provide redundant functionality at less cost 
than other information tools. At best, QSP may meet the first condition, depending on the 
specific needs of each school. Namely, QSP may provide schools with the unique ability 
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to integrate local data with district data. Also, QSP may provide unique support for 
classroom-level analysis through its graphing and reporting functions, especially the 
analysis of local data. 

As for the second condition, that of supporting data-driven decision-making 
processes more effectively than competitive information tools do, QSP probably does not 
succeed. Table 4 represents key functions considered important in data-driven decision- 
making for MPS employees. In each column, a check appears if the software provides 
support for that function. A check-minus indicates either that the software does not 
support that function very well (i.e., technically inadequate), or that there are usability 
problems (i.e., poorly implemented). Conversely, a check-plus indicates that the software 
does a good job at supporting that function, whereas a check indicates that the software 
meets the basic criteria for that function. An “X” indicates that the software does not 
support that function. Assigning 0 for each “X”, 1 for a check-minus, 2 for a check, 3 for 
a check-plus, and taking the average, it is possible to calculate an overall rating. If certain 
functions are considered more important than others, those can be given a higher 
weighting in the overall score. The last column in Table 4 lists the weightings that we 

Table 4 



Comparative Analysis of Key Data Functions 





QSP 


Excel 


Brio 


SMS 


Weighting 




Exporting Data 










5 




Importing District Data 








X 


10 




Importing Local Data 






X 


X 


10 




Query Data Warehouse 


X 


X 




X 


7 




Sorting 






'/F 




10 


53 


Grouping 










10 


.2 


Pivot Tables 


X 






X 


7 


e 

:3 


Individual Student Reports 










10 




Bar Charts 








X 


7 




Pie Charts 


>/'+ 






X 


7 




Tables 






v+ 


X 


7 




Cross-tabular Tables 






'/F 


X 


7 




Editing Graphs 


X 






X 


7 




Statistical Analysis 








X 


5 




Equally Weighted Average 


1.9 


2.2 


2.4 


0.2 






Weighted Average* 


2.0 


2.2 


2.6 


0.3 





Note: The overall rating was calculated by assigning a value of 3 for each 2 for each , 1 for 
each -, and a 0 for each X. 
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have assigned to each function and the last row in the table reports the weighted average 
using those weightings. As can be seen, QSP and Brio both score a little higher with the 
weightings in place, but the relative scores of all the tools remained the same. 

Table 4 further shows that QSP does not support any function that is not also 
supported by either Brio or Excel. In addition, there are two functions that QSP supports 
poorly, importing data and statistical analysis. This does not mean that QSP cannot be a 
viable and useful tool in MPS, but it does present some challenges. QSP might still fill a 
niche in some MPS schools that have a high need for importing local data, working with 
“sortable” groups of that data, and having a low need for editable graphs. Also, QSP 
provides reporting functions for individual students. However, Brio could also be used to 
generate similar reports, though doing so would probably require greater expertise than 
QSP. Likewise, Excel can leverage the scripting language of visual basic to generate 
batch reports, but again, doing so requires a high level of expertise. 

For example, QSP may be useful in schools for looking at in-house data that are 
not in the Data Warehouse. The key to this will lie in the degree of difficulty encountered 
in importing data into QSP. For example, two of our schools were interested in importing 
test data from the Renaissance Learner’''^ STAR MATH® and STAR READING® 
software (see http://www.renleam.com/default.htm) . However, this software does not 
support the exporting of data into tab- or comma-delimited text files. Therefore, we 
helped these schools develop a way of exporting data that involved printing data to a 
post-script file.^ This method required using Excel to clean up extraneous information, 
such as page headers and misaligned columns. Overall, the process involved importing 
from the STAR software into Excel and then into QSP. Some schools reported that it was 
easier to use Excel only. 



Summary and Conclusions 

Milwaukee Public Schools provides a complex and advanced technological 
environment in which to study the impact of QSP on data-driven decision making. The 
nature of information flow within MPS presents schools with several challenges in 
working with data. District data sets are large, and data are often available in non-flat file 
format only, requiring users to posses sophisticated computer skills to make the files 
transferable and usable. In addition, MPS information system hardware and software are 
still under development and implementation, and the value of district data is still being 
investigated and developed. Given this complex technological environment, this report 
has tried to summarize the major benefits of QSP and how those features interact with the 
technology already in place in MPS. The beneficial features of QSP — its capacity for 
importing data, grouping data, and graphing data — are in fact very important to data- 
driven decision-making. It is at the school level that QSP best meets the needs of users. 
However, as demonstrated throughout this report, many of these features can be 
replicated in either Excel or Brio. Furthermore, Excel and Brio are required in many data 
activities for MPS schools. Brio is needed for accessing data stored within the data 



^ A post-script file is a file that contains special formatting instructions. These are most often associated 
with either a printer file or an adobe .pdf file. 
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warehouse, whereas Excel is needed for manipulating and cleaning local data sets. Both 
of these tools are useful in flattening or cleaning data sets so that they can be analyzed 
more easily. 

The four-dimensional framework is useful in understanding how various 
information tools compare and contrast to each other. It is also useful in describing or 
categorizing information tools, as well as in guiding more detailed comparisons. This 
framework appeared to be particularly useful when demonstrating how QSP differs from 
Excel, Brio, and SMS in terms of meeting user needs, analytical structures, and technical 
considerations. Data characteristics are primarily determined by factors other than the 
end-user tool, but even so, the data characteristics of a data set can affect the goodness of 
fit for a particular tool when that tool has limited capacity. Overall, the framework proved 
useful in establishing a model for a contextual analysis of QSP. 

The diffusion of QSP within MPS will be limited by three factors. First, there are 
few features provided by QSP that are not also supported by either Excel or Brio. This 
minimizes the necessity of adopting QSP as a regular information tool. In addition, QSP 
ranks, at best, as no more effective than Excel and less effective than Brio in how well it 
supports the basic functions related to data use in MPS. Finally, some of the problems 
associated with QSP’s import wizard and graphing modules may require users to use 
either Excel or Brio. Ideally, the use of QSP should lead to the decreased use of other 
tools. 



However, QSP may yet be useful to schools that place a high value on working 
with local data, or that have solved many problems relating to importing data into QSP. 
This would be much more likely in smaller school districts where test data files are not so 
massive, which would minimize many of the problems that MPS schools reported while 
trying to import data. In addition, QSP may find a niche in schools that place high value 
on producing individualized student reports. 

In general, there are major issues to be confronted if we are to make the process 
of data-driven decision making more efficient. First, schools must have access to useful 
data that are formatted in a way that encourages data analysis. Schools must also learn 
how best to manage their data and information tools so that resources are maximized. 
Finally, schools must learn how to utilize data as a feedback signal so that school and 
classroom practices are improved over time. These problems are not entirely independent 
of the problems relating to QSP. QSP is, in fact, designed in a way that assumes that the 
user has access to clean and meaningful data. Therefore, it is not surprising that problems 
were encountered when schools tried to use QSP on data that was not formatted in 
exactly the right way. Indeed, schools that demonstrated the greatest success were the 
schools that devoted the most personnel to the problems associated with using data and 
that spent the greatest amount of time seeking meaningful pieces of information and 
knowledge in that data. 
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